Purpose: To use a pre-trained model that finds a hand in a live webcam picture, returns its keypoints as coordinates and use those coordinates in our game logic

Exercise (1)

• Use the ml5.js hand tracking model to detect the index finger and mark its position on the canvas.
• Click here to download the base code to start from
• Click here to view the hand and finger keypoints
• Reference: the ml5.js HandPose model documentation.

Exercise (2)

• Part 1: Create a program that lets users draw in the air using their index finger.
• Part 2: Extend the program to detect and track multiple hands at the same time.

Solutions

• Link to view the solution for Exercise 1
• Link to download the solution code for Exercise 1
• Link to view the solution for Exercise 2 Part 1
• Link to download the solution code for Exercise 2 Part 1
• Link to view the solution for Exercise 2 Part 2
• Link to download the solution code for Exercise 2 Part 2

1. Check that the webcam works and note that ml5.js loads the model from the internet, so the internet connection is needed.
2. Start from the base code linked in Exercise 1. It already has the variables and the empty steps, so the students fill in the model loading and the drawing rather than typing the whole program.
3. Explain that this model is pre-trained - nobody in the class trained it. The program sends it a picture and it sends back a list of points.
4. Show the keypoints image linked in Exercise 1. Draw out that the model returns 21 numbered points per hand, not the word “hand” - point 8 is the index fingertip, which is why the code picks that one.
5. The results come back as an array, one entry per hand detected by the model. Detecting several hands is the same code reading more entries from that array.
6. Discuss where gesture control is already used - games, phones, cars, even electric chimneys at home and malls.
7. Points to Ponder:
   • We did not train this hand model. Somebody else did. What kind of pictures do you think they would have used to train it? Would it be supervised training or unsupervised training?
   • Where all do you think the hand model can be used in the real world? Discuss. Where would touch-free control be genuinely useful and where would it be worse than a button?